The present invention relates to video compositing and, more particularly, to a method for providing a background pattern so that the position and orientation of a moving camera can be determined in real time for the realistic compositing of images of a foreground object with a virtual background.
The technique of "chroma-key" compositing has long been used in video production to combine images of a foreground object with a virtual background stored in a digital data base. In this technique, the foreground object is photographed against a background of a "chroma-key" color, typically a particular shade of blue. In the digital images thus captured, all the pixels colored with the chroma-key color are replaced with pixels derived from the virtual background. In order for the background pixels to be rendered in a way that provides a realistic composite image, the position and orientation of the camera must be known.
Several method are known in the art for determining the position and orientation of the camera. These methods can be classified in two categories. In methods of the first category, the position and orientation of the camera are tracked explicitly. In these methods, the camera must be provided with special tracking devices that are rigidly attached thereto. In some of the methods of the first category, the tracking devices are encoders, and lateral movement is tracked by encoders that are in physical contact with the floor of the studio. Another example of the methods of the first category, a magnetic tracker system, is taught by Loftus et al., in PCT application no. US96/04846. Loftus et al.'s special tracking device is a magnetic tracker receiver rigidly attached to the camera. In the methods of the other category, the position and orientation of the camera are inferred from the captured images. For example, Graham, in U.S. Pat. No. 5,502,482, which is incorporated by reference for all purposes as if fully set forth herein, teaches the use of a chroma-key background of two or more shades of blue in a predetermined pattern, for example a checkerboard pattern. The position and orientation of the camera are inferred from the locations of features of the pattern on the captured images.
Methods such as Graham's have the advantage of allowing the use of conventional video cameras, without special tracking equipment. This is particularly convenient for video compositing with hand-held cameras, to avoid the necessity of contacting lateral motion encoders with the floor of the studio, or to avoid the extra bulk and weight of tracking equipment such as that of Loftus et al. Nevertheless, these methods have limitations of their own. One limitation is that the pattern is fixed, in both level of detail and location. If the camera zooms in too closely on the foreground object, there may be too few features in the portion of the image occupying the camera's field of view for the location and orientation of the camera to be determined accurately. Conversely, if the camera is too far from the background pattern, the pattern features may be sufficiently crowded to make it difficult to distinguish between a panning motion of the camera and a lateral translation. In principle, the pattern can be provided with multiple levels of detail, to preserve tracking resolution at all required distances; but this adds to the complexity of the pattern. Similarly, if the foreground object is a live actor who moves laterally with respect to the pattern, and the camera follows the actor, the actor may move so far laterally that not enough of the pattern is left in the camera's field of view to allow the camera's location and orientation to be determined accuracy. A second limitation is that the multiplication of chroma-key colors has been found to degrade the realism of the composite image. The narrower the spectral band of the pixels that are replaced with virtual background pixels, the more realistic the resulting composite image. The loss of realism associated with the use of a multi-shade chroma-key background persists even while the camera is in a fixed position and only one chroma-key color would suffice. A third limitation is that the pattern is fixed in place in the studio, typically being painted on a wall or a fixed partition. This limits the flexibility of the method and precludes its use, for example, in remote locations.
There is thus a widely recognized need for, and it would be highly advantageous to have, a method for providing an image for dynamic chroma-key compositing that is free of the limitations of the methods known in the art.